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PROMOTING QUALITY THROUGH 




Harvard Family Research Project’s (HFRP) Issues and 
Opportunities in Out-of-School Time Evaluation briefs 
highlight current research and evaluation work in the 
out-of-school time field. These documents draw on 
HFRP’s research work in out-of-school time to provide 
practitioners, funders, evaluators, and policymakers with 
information to help them in their work. Recognizing the 
critical role that staff play in promoting quality out-of- 
school time ( OST) programs, in this brief we examine 
OST professional development efforts and offer a frame- 
work for their evaluation. 

T here is no question that staffing is a key component 
of quality in out-of-school time (OST) programs.' 
Many programs attribute their success to skillful 
providers, and research demonstrates the importance 
of positive staff-child relationships for youth outcomes 
(Eccles & Gootman, 2002; McLaughlin, 2000; Rosenthal 
& Vandell, 1996; Tolman, Pittman, Yohalem, Thoma- 
ses 8c Trammel, 2002). Recently, when a set of leading 
experts in the OST field was asked to identify the single 
most important ingredient for creating and sustaining 
quality improvement systems in OST, five of the eight 
respondents articulated issues of staff recruitment, train- 
ing, and development (Little, 2004). 

Professional development initiatives have been a key 
mechanism for increasing the skills of everyone from new 
personnel to longtime staffers, in part because many OST 
workers are paraprofessionals who have little preservice 
training (Partnership for Afterschool Education, 1999). 
However, evaluating the effectiveness of these initiatives is 
a relatively recent development. Given the importance of 
staff development for both higher quality programs and 
better youth outcomes, it is critical that we design evalu- 
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ations that will help programs understand the benefits of 
their professional development efforts. 

This brief offers a preliminary framework for under- 
standing and implementing evaluations of OST profes- 
sional development initiatives. In so doing, it draws on 
research from early childhood and elementary education 
and highlights the evaluation methods and results of re- 
cent OST professional development initiatives at both 
local and national levels. 2 



Professional development is a broad term that can refer 
to a variety of education, training, and development 
opportunities. For the purpose of this brief, the term 
will be applied to a full range of activities that have the 
common goal of increasing 
the knowledge and skills 
of staff members and vol- 
unteers. Professional devel- 
opment programs target 
many specific outcomes, 
including increased staff 
knowledge about child and 
adolescent development, 
use of effective strategies 
for activity programming, 
and implementation of 
methods for promoting 
positive relationships with 
youth. Many professional development initiatives also 
have the secondary goal of improving the quality and 
sustainability of the out-of-school time workforce, by 
increasing providers’ marketable skills and by garnering 
public support for the youth development field. 

Professional development for OST providers can oc- 
cur in many settings and at many stages throughout a 
provider’s career. These activities may be formal or casual, 
highly structured or flexible, conducted before the job 
application process or throughout the career cycle. Pro- 



What Is Professional Development! 



Given the importance of 
staff development for both 
higher quality programs and 
better youth outcomes, it 
is critical that we design 
evaluations that will help 
programs understand the 
benefits of their professional 
development efforts. 
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DEFINING ESSENTIAL SKILLS 
FOR YOUTH WORKERS 

An increasing number of stakeholders are asking the 
question, what defines a skillful out-of-school time 
provider? The answer is a complex one, particularly 
given the diverse experiences, responsibilities, and 
needs of youth workers. 

One way to address the issue is through youth worker 
competency standards. Several organizations have de- 
veloped competency frameworks, and a framework 
from the National Collaboration for Youth aims to 
integrate these into one unified model (see Select 
Professional Development Resources on page II). 
Competency standards come from several sources, 
including the voices of youth workers themselves, 
research from the fields of early childhood education 
and daycare, standards from the U.S. Department of 
Labor on successful workplace skills, developmental 
research on the needs of youth (e.g., the National 
Research Council’s publication Community Programs 
to Promote Youth Development ), and, most recently, 
youth reports (e.g., qualitative research by Reed 
Larson from the University of Illinois at Urbana- 
Champaign). 

While no clear consensus yet exists, several fac- 
tors have been consistently identified from these 
sources, including, but not limited to, possessing a 
strong knowledge of youth development; engaging in 
warm, positive relationships; promoting positive peer 
interactions; and encouraging youth to be actively 
involved in shaping their programs and experiences. 
Identifying other areas of overlap is an important area 
for future work, in order to provide a comprehensive 
picture of a skillful provider. 

fessional development opportunities are diverse largely 
because OST workers represent a range of backgrounds — 
there is no standard route to becoming a provider. A large 
number of staff who work directly with youth have no 
preservice training and many programs do not require 
specific credentials or academic degrees; in fact, very few 
colleges and universities offer courses or degrees in OST 
or after school (Partnership for Afterschool Education, 
1999). Therefore, a range of professional development 
opportunities can be beneficial, and overarching “profes- 
sional development systems,” or combinations of various 



development activities, are likely to show the largest and 
most sustained effects when they are evaluated (Costley, 
1998). Professional development settings include: 

■ Higher education, such as continuing education 
courses and degree programs 

■ Preservice training and new-staff orientation 

■ In-service training provided by programs to current 
staff 

■ Training seminars and resource centers provided by 
external organizations outside the program setting 

■ Local and national credentialing systems and pro- 
grams 

■ Local and national conferences 

■ Mentoring programs and relationships 

■ Ongoing informal resources, such as newsletters, 
online discussion boards, and “brown bag” lunches 
for staff members to share ideas and expertise 

In many national organizations (e.g., Girls Inc. and 
4-H), staff development is coordinated by a central 
office and integrated into the national infrastructure. 
Lor smaller local programs, access to professional de- 
velopment opportunities is often facilitated by regional 
“clearinghouses” and advocacy organizations. A recent 
and promising approach to professional development 
is to harness audio, video, and computer technologies 
to provide distance-learning programs for teacher and 
staff training. 

Lrom college courses to staff training seminars, profes- 
sional development activities vary according to program 
characteristics and providers’ needs. At the program 
level, operating budgets and program infrastructure 
have a large impact on the kinds of trainings and other 
opportunities offered. In addition, the type and setting of 
professional development activities depend, in part, on 
the program’s staffing model (i.e., who is hired and for 
what purpose). OST staffing models range from employ- 
ing full-time directors who harness volunteer energy, to 
engaging older youth as staff members, to contracting 
with content-specific consultants to work on special arts 
and music programs. Many programs incorporate several 
of these methods. 

Lor example, the New York City Beacons Initiative 3 
offers a career ladder, which can eventually lead from the 
volunteer level to paid employment. The program encour- 
ages youth to run many activities themselves and to be- 
come future providers, offers the MOSAIC Employment 
Club to provide job skills training to community mem- 
bers, employs only college graduates or current students 
to promote higher education for youth participants, and 
holds mandatory weekly training meetings. LA’s BEST 4 
provides opportunities for “alumni” to return to the pro- 
gram as volunteers and eventually as paid staff members, 
thus continuing to engage young people in the program 
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even after they have “graduated.” The San Francisco 
Beacons Initiative 5 also offers a variety of training mod- 
els for their diverse staff, including a citywide training 
network for directors and administrators, weekly training 
for Americorps volunteers, and on-the-job training for 
contractual employees, such as visiting artists. 

How Does Professional Development 
Impact Program Quality? 

Professional development activities can impact program 
quality on at least two levels. First, and most important, 
staff development can affect youth outcomes. Evidence 
of this process comes from other fields related to OST. 
Early childhood education studies have found that the 
quality of daycare children receive is associated with 
providers’ educational attainment and participation in 
training workshops (NICHD Early Child Care Research 
Network, 2001; Norris, 2001), and that quality of care 
is in turn linked with children’s more positive social and 
cognitive outcomes (NICHD Early Child Care Research 
Network, 2001). Similarly, an education reform study 
showed that elementary school students performed higher 
in reading and math when their teachers had participated 
in and given high ratings to a professional development 
program that provided content knowledge and instruc- 
tional strategies aligned with state education standards 
(Westat & Policy Studies Associates, 2001). 

The positive impact of staff development most likely 
operates through a chain of events: Professional devel- 
opment activities affect the practices that youth workers 
use in their programs, and these practices in turn affect 
youth participants. For example, well-trained providers 
use their knowledge of child and adolescent develop- 
ment to interact positively and effectively with youth. 
Research from developmental psychology has shown that 
having at least one positive and caring relationship with 
an adult can buffer children against risk factors in their 
lives, and in fact, can predict positive outcomes in later 
life (e.g., Gambone, Klem & Connell, 2002; Werner & 
Smith, 1982). OST programs provide a vital opportunity 
for such connections, whether the program’s focus is on 
academics, leadership skills, or other outcomes (Seligson 
& Stahl, 2003). Staff training can also help providers plan 
enriching activities, connect with participants’ families, 
and communicate with community stakeholders — all of 
which are elements of high quality programs (Eccles & 
Gootman, 2002; Rosenthal & Vandell, 1996; Tolman 
et ah, 2002). 

Secondly, professional development is important be- 
cause of its potential impact on the sustainability of the 
youth development workforce. OST providers who have 
professional development opportunities have reported 
feeling more confident and more satisfied with their jobs 
(Center for School and Community Services, Academy for 



Educational Development, 2002; Girls Inc., 1996). These 
outcomes could lead to better staff retention and lower 
turnover rates. It is important to emphasize the possibil- 
ity, however, that staff retention may improve only if staff 
who have training opportunities also have opportunities 
for increased responsibility and higher compensation. 

One method for addressing this need is the career 
development ladder, a system that would tie increases in 



THE APPRENTICESHIP MODEL 

A new and promising approach to professional devel- 
opment is the apprenticeship model, in which youth 
organizations provide employment in out-of-school 
time settings, along with training and college-level 
courses. This model benefits both staff and programs: 
It provides staff members with training and experi- 
ence, which will increase their skills and job opportu- 
nities, and provides programs with dedicated staffers 
who are committed to youth development. 

The Youth Development Practitioner Appren- 
ticeship Program (YDPA ) 1 is a national initiative 
of the U.S. Department of Labor (DoL) to provide 
a special out-of-school time certification for current 
and future youth workers. Local YDPA programs 
combine mentoring, direct instruction, and on-the- 
job training, on topics ranging from youth develop- 
ment to program organization. DoL is working with 
organizations around the country, including 4-H, 2 to 
implement this initiative. 

The Teaching Fellowship Program 3 is another 
apprenticeship model, run by Americorps, in partner- 
ship with Boston’s Citizen Schools, Lesley University, 
and local nonprofit and commercial organizations. 
Participating Teaching Fellows work at local organi- 
zations in the mornings, teach at Citizen Schools in 
the afternoons, and work toward a master’s degree 
in education with a special concentration in Out-of- 
School Time Education. 

1 For more information on YDPA, visit the National Clearinghouse 
for Youth Development Practitioner Apprenticeship Programs 
website at www.ydpaclearinghouse.org. 

2 For more information on the National 4-H Youth Develop- 
ment Practitioner Apprenticeship Certificate Program, visit www. 
nae4ha.org/ydpa. 

3 For more information on the Teaching Fellowship Program, 
visit the Citizen Schools website at www.citizenschools.org/ 
teachingfellows/index.cfm. 
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PROFESSIONALIZING THE WORKFORCE: 
WASHINGTON STATE’S 
CAREER LADDER INITIATIVE 

The Washington State Early Childhood Educa- 
tion Career and Wage Ladder is a pilot project 
run by the Economic Opportunity Institute. It uses 
public funding to tie salary increments for childcare 
workers to continuing education, experience, and 
responsibility. Its goals are to professionalize the 
status of childcare workers, increase job satisfaction 
and retention, and improve the quality of childcare. 
Evaluation results from the pilot study showed that 
childcare workers in centers with career ladders had 
higher job retention rates, exhibited higher morale, 
and were more likely to complete higher education 
courses than childcare workers in centers without 
career ladders. For more information on this proj- 
ect and its results, visit www.eoionline.org/Policy- 
EarlyLearningAndCare.htm#ECE. 



staff salaries and responsibilities to ongoing education 
and experience. Advocates propose that a career devel- 
opment ladder would increase the professional status of 
OST providers and, correspondingly, staff retention rates 
(Brown, 2002). Findings of a recent pilot study indicate 
that a wage ladder in early childhood education had 
positive effects on staff retention and morale (see the 
box on this page). Although there are some distinctions 
between early childhood and OST (most obviously, the 
age of the children served), early childhood education 
initiatives may serve as models for the OST field, and 
there is some evidence to suggest that including a career 
ladder is a key component for system building in OST 
(National Institute on Out-of-School Time & Academy 
for Educational Development Center for Youth Develop- 
ment and Policy Research, 2003). 

Why Evaluate Professional Development 
Initiatives! 

Despite the prevalence of professional development 
activities and their numerous potential benefits, the ef- 
fectiveness of these efforts cannot be ensured by good 
intentions alone. Evaluations are needed to measure and 
verify their quality and usefulness. However, evaluations 
of professional development have been sparse. According 
to a scan of our Out-of- School Time Program Evaluation 
Database, 6 very few program evaluations include infor- 
mation about training or staff development, and staffing 



information in these evaluations is usually limited to 
hiring qualifications and staff/child ratios. This could be 
due to a number of barriers, including limited time and 
resources for evaluation and limited evaluation skills. 
Educators and other youth workers often report feel- 
ing that evaluation of professional development wastes 
valuable staff time (Guskey, 2000). However, evaluation 
is a critical part of the professional development process 
because it identifies which program elements are (or are 
not) successful, which ultimately leads to the creation of 
more effective and efficient programs. 

Although many core features of assessment are 
consistent across contexts, there are some features of 
evaluating professional development programs and 
initiatives that are unique. Most important, and unlike 
many other program elements, it is necessary to demon- 
strate the effect of professional development activities at 
multiple levels of outcomes. The next section of this brief 
describes a multilevel framework for understanding and 
implementing evaluations of professional development 
in OST programs. 

A Multilevel Framework for Evaluation 

Professional development experts agree that evaluations 
can and should assess the effects of professional develop- 
ment programs at multiple levels (Guskey, 2000; Killion, 
1998; Kirkpatrick, 1998; Mizell, 2003). Some experts rec- 
ommend three levels, while others add a fourth. Kirkpat- 
rick’s framework, which has been a model for evaluating 
training initiatives for several decades, details four levels 
of evaluation: (1) reaction to the training, (2) learning of 
information and practices from the training, (3) transfer 
of this knowledge into practice, and (4) results for key 
stakeholders. Table 1 below presents these four levels and 
applies them to professional development in OST. 

As noted above, the reality of evaluation in the con- 
text of limited time, resources, and evaluation skills often 
translates into a scarcity of much-needed evaluation in- 



TABLE I 

THE FOUR-LEVEL EVALUATION MODEL 
APPLIED TO OUT-OF-SCHOOL TIME (OST) 



Level 


Applied to OST 


1 . Reaction 


Feedback from OST providers about training 
needs and satisfaction 


2. Learning 


OST providers’ knowledge of best practices 


3. Transfer 


The practices used by OST providers 


4. Results 


Positive developmental outcomes for youth 
and other stakeholders, such as families and 




communities 
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formation that programs can use for accountability and 
program improvement. The four-level model can facilitate 
professional development evaluations in OST, because it 
describes how evaluation can be conducted — and how it 
can be useful — at each level. 

Before planning an evaluation strategy or choosing an 
evaluation level, it is important for professional develop- 
ment evaluators to define their goals. These goals may be 
affected by the program’s stage of development. For ex- 
ample, evaluation of a first-time program may place more 
emphasis on implementation and provider opinions; after 
implementation has been studied and improved, evalu- 
ators may then turn more attention to youth outcomes. 
Each of the four evaluation levels plays an important 
role in facilitating learning about the initiative and, ulti- 
mately, in improving it. The sections that follow describe 
the uses of each level, present examples of evaluations at 
each level, and, finally, offer recommendations for future 
evaluation. 

Our review of evaluations reveals that most current 
professional development evaluations focus on the first 
level: They collect information about provider needs 
and program satisfaction that can be used to reflect 
on program practice. Only a few program evaluations 
have reported outcomes on the second and third levels 
of provider knowledge and program practices, and, to 
our knowledge, no completed studies have yet examined 
the fourth level — how professional development efforts 
impact participant outcomes. The results of our scan of 
OST professional development evaluations and their 
findings are provided below and are also compiled in 
Table 2 on page 8. 

Level I - Reaction: Understanding Providers’ 
Needs and Satisfaction 

The first level of evaluation is quite common among OST 
professional development initiatives: Many initiatives de- 
scribe the demographics of participating youth workers 
and their satisfaction with the programs. The question 
answered by these surveys relates to staff members’ per- 
ceptions of the training programs. As such, the results 
can be helpful in improving the professional development 
program from the perspective of its participants, which 
could lead to higher participation and higher levels of 
engagement. 

Level 1 evaluations usually take the form of brief par- 
ticipant surveys, either immediately following the training 
or soon after. For instance, the After- School Corporation 
(TASC) 7 studies perceptions of their professional devel- 
opment programs from multiple perspectives, including 
feedback from the participants at the conclusion of the 
workshops, self-assessments by the contractors who run 
the trainings, and annual surveys of site coordinators 
at each OST program. Similarly, Bridging the Gap, 8 a 



training program run by Boston Public Schools to link 
OST learning with school learning, administers surveys 
of perceptions about the training’s usefulness immediately 
after the program and also later in the year for follow-up. 
Both organizations report that this feedback has helped 
them to better understand the needs of OST providers 
and to improve their training initiatives. 

Levels 2 and 3 - Learning and Transfer: 
Increasing Knowledge and Improving 
Program Practices 

Some evaluations conducted at both the state and na- 
tional levels have taken additional steps to document the 
effects of professional development on the second and 
third outcome levels — OST provider learning and transfer 
of this knowledge into practice. The learning and transfer 
categories can refer to both general principles, such as 
youth development and effective management, and to 
specific content, such as the application of a lesson or ac- 
tivity that was taught during a professional development 
workshop. Level 2 and 3 evaluations can help establish 
the effects of professional development initiatives on OST 
providers, providing valuable information about whether 
the initiatives should be continued, how they work, and 
how they can be improved. Below we feature five efforts 
to examine increased knowledge and improved program 
practices through evaluation levels 2 and 3. 

Making the Adost of Out-of-School Time 
Making the Most of Out-of-School Time (MOST) 9 was 
a multiyear initiative, designed to build community-wide 
support and infrastructure for OST programs in Boston, 
Seattle, and Chicago. As part of its system-building ef- 
forts, MOST supported the creation and availability of 
college-level courses, tuition subsidies, staff mentoring, 
on-site training, and other professional development 
activities. These professional development efforts were 
evaluated with a descriptive methodology, using observa- 
tions, interviews, and surveys to collect information at 
participating programs (Halpern, Spielberger & Robb, 
2001). Specifically, evaluators examined a variety of OST 
program features, including staff training and quality. 

Evaluators of the MOST initiative found that the staff 
development efforts were effective, but more so for some 
staffers than others: Program directors benefited more 
than “front-line” providers. For front-line staff, work- 
shops and college courses increased their knowledge of 
youth development, but this knowledge was not often 
integrated into their program practices. Furthermore, 
there were particular barriers to the effectiveness of col- 
lege course offerings: Staff members were reluctant to 
enroll, frequently did not complete courses, faced time 
and financial constraints, and often found that course 
completion did not result in higher salaries. 
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San Jose 4 Quality 

A similar methodology was used to evaluate San Jose 4 
Quality, 10 a multifaceted initiative for improving OST 
programs in the San Jose, California, area. San Jose 4 
Quality was designed to affect four desired outcomes, 
including creation of a better-trained workforce in lo- 
cal programs. Staff improvement components included 
four staff trainings per year with follow-up meetings and 
presentations to discuss and reinforce implementation 
of training material, regular meetings of lead staff from 
local programs, and site-specific technical assistance. The 
initiative has been evaluated using a range of methods, 
from observations to interviews to surveys, with findings 
due out in an annual report in 2004. 

Like MOST, preliminary results of San Jose 4 Qual- 
ity suggest a pattern of positive results, accompanied by 
some challenges. Notably, staff members reported that the 
initiative broadened their horizons, heightened their sense 
of professional affiliation, and increased opportunities for 
networking and support. Other results of the initiative 
may also be related in part to the staff training compo- 
nent, such as improved staff interactions with youth. 

Girls Inc. Strong, Smart and Bold Initiative 
An evaluation of Girls Inc.’s Strong, Smart and Bold 11 
initiative used baseline data 12 to move the field toward 
a higher level of rigor (Girls Inc., 1996). The initiative 
provided intensive training for program directors in de- 
veloping skills for planning, implementing, managing, 
and evaluating best practices, and in integrating the Girls 
Inc. mission at local affiliate programs. The training was 
grounded in a combination of theories of youth devel- 
opment, organizational change, and adult learning and 
was envisioned as a systems approach in which program 
directors and executive directors worked together. 

The initiative was assessed using two types of evalua- 
tions. An internal evaluation conducted by Girls Inc. staff 
was used primarily for continuous program improvement, 
while a case study evaluation of four affiliate programs 
was conducted by external evaluators. The case study 
evaluation used a qualitative methodology 13 to assess or- 
ganizational culture, operations, and programs of four 
representative affiliates. Data were collected from site 
visits at four time points, using semi-structured interviews 
(with directors, staff, youth participants, their parents, 
and community stakeholders), structured program ob- 
servations, and site “portfolios” of program records and 
self-assessments. 

The evaluation illuminated many positive effects at 
evaluation levels 2 and 3, including (but not limited to) 
increased confidence, enthusiasm, and understanding of 
Girls Inc.’s mission among program directors; increased 
capacity to serve traditionally underserved populations; 
implementation of more varied programming and increas- 



es in youth participants’ active involvement in program 
planning; and directors’ use of more intentional planning 
strategies in both the short and long term. 

However, the evaluation also revealed some less posi- 
tive findings. Most notably, staff turnover remained a 
significant challenge for the affiliate programs after 
the initiative, and the goal of fostering teamwork be- 
tween program directors and executive directors was 
not always fulfilled. In addition, although respondents 
reported increased awareness of the importance of pa- 
rental involvement, diversity in staff recruitment, and 
ongoing staff training, little progress was made in these 
areas. Results revealed that front-line staff needed more 
ongoing training in youth development and appropriate 
activity structure. 

North Carolina Quality Enhancement Initiative 
Baseline comparisons were also used in a study evaluat- 
ing the North Carolina Quality Enhancement Initiative 
(NC QEI) (Hall & Cassidy, 2002), a project designed to 
prepare selected OST programs in North Carolina for 
the process of accreditation by the National School-Age 
Care Alliance (NSACA) (now known as the National 
Afterschool Association). Like the MOST Initiative, NC 
QEI was a comprehensive program improvement project 
that included a professional development component. The 
initiative began with a 2-day staff workshop and provided 
ongoing consultation and support for 9 months. 

Both before and after the initiative, standardized ob- 
servations and staff interviews were conducted to obtain 
quantitative data on OST program environments and 
staff-child relationship quality. Participating programs 
showed statistically significant improvements in each of 
these areas. Furthermore, the largest improvements were 
in programs that initially scored lowest on a predeter- 
mined set of quality indicators, i.e., those which served 
larger numbers of youth, did not have state licenses, and 
were led by directors without college degrees. One caveat 
should be noted, however: It is possible that program 
improvements were due not only to the training program, 
but also to awareness of and increased effort for the ensu- 
ing accreditation process. 

Building Exemplary Systems for Training Youth Workers 
An evaluation of the Building Exemplary Systems for 
Training Youth Workers (BEST) initiative 14 also re- 
ported statistically significant changes from baseline to 
follow-up at outcome levels 2 and 3 (Center for School 
and Community Services, Academy for Educational 
Development, 2002). To address the problem of frag- 
mented, disjointed training efforts, BEST was created as 
a cohesive and theory-based approach to professional 
development. The primary goal of the program was to 
provide staff training, with a secondary goal of building a 
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community-wide career support infrastructure for youth 
workers (e.g., availability of college degree programs, 
accreditation systems, etc.). While the 15 national sites 
differed in minor ways, all relied heavily on a positive 
youth development curriculum, which emphasized the 
strengths and assets of youth. 

Built in to the BEST initiative was an evaluation study 
that used surveys, interviews, and diaries administered 
before and after the program to assess whether the train- 
ing program changed the practices of participating youth 
workers. Using this method, researchers found that, for 
the majority of youth workers (over 70%), BEST had a 
positive impact on the practices they used in their pro- 
grams. Specifically, data indicated the following signifi- 
cant changes after the training: Youth workers (1) were 
more likely to integrate youth development knowledge 
and theory into their program planning, (2) were more 
likely to involve youth participants in the process of 
running the program, (3) learned a common language 
of positive youth development, which allowed them to 
communicate more effectively, and (4) experienced a 
greater amount of networking and information sharing 
with other providers. 

The BEST program also partially achieved its second 
goal: Youth workers and community stakeholders re- 
ported an increase in the amount of support provided 
by their organizations in the form of staff mentoring 
and supervision. Elowever, youth workers continued to 
request more support after the program. In interpreting 
these results, it is important to note that the BEST study 
assessed training participants’ reports of whether they 
changed their youth work practices, and did not include 
more objective measures such as observations by external 
evaluators. 

The level 2 and 3 evaluations reported here demonstrate 
both the potential of staff development initiatives and the 
need for additional strategies and investments to translate 
this knowledge into practice. A common theme across all 
the evaluations was that the staff initiatives were beneficial 
but not sufficient on their own, particularly for front-line 
youth workers. These studies also offer important lessons 
about evaluation design: Each evaluation approach had 
unique strengths (e.g., MOST provided rich descriptive 
information, BEST used baseline comparisons, Strong, 
Smart, and Bold used program observations), but none 
combined these methods for the most rigorous possible 
evaluation design. 

Level 4 - Results: Assessing Impact on 
Participants 

The studies described above represent important advances 
in evaluation. However, they do not demonstrate the im- 
pact of professional development initiatives on youth out- 



comes. Clearly, level 4 evaluations play an essential role 
in demonstrating the ultimate goal of all staff work — to 
improve the lives of youth. These evaluations can also 
play an important role in supporting sustainability for 
professional development initiatives. However, program 
directors and evaluators should be aware that positive 
youth outcomes, while most conclusive, are also the most 
difficult to establish. A multitude of factors operate to 
influence youth outcomes, both within and outside of 
OST programs. Furthermore, OST programs aim to affect 
a broader range of outcomes and operate on a less struc- 
tured and defined schedule than school-day programs. 
The complexity of these features makes it difficult to 
demonstrate that youth outcomes are due specifically to 
staff training. Evaluators should also consider the age 
and stage of their professional development initiatives; 
affecting youth outcomes takes time, and newer programs 
may wish to first investigate levels 1, 2, and 3. For these 
reasons, it is important to examine youth outcomes in 
combination with the other three levels of outcomes 
(satisfaction, knowledge, and practices). 

To date no studies have reported on the fourth out- 
come level — whether professional development programs 
affect outcomes for the youth served by participating 
OST providers. Level 4 outcomes will be part of an in- 
progress evaluation of the New York State School-Age 
Care Credentialing Project (SACC). SACC, begun in 
1999, promotes quality youth services by providing spe- 
cific standards, training, and evaluation of school-age care 
providers. In 2002, the Cornell Early Childhood Program 
began to evaluate this new statewide credential, using a 
multistage evaluation approach similar to the framework 
we present in this brief. 15 

To date, this multifaceted evaluation has focused on 
three main goals. The first goal is to document the history 
and development of SACC through in-depth interviews 
with SACC development leaders. The second goal is to 
examine the effects of SACC on the following level 1 
and 2 evaluation outcomes: school-age care providers’ 
participation in professional development; beliefs and 
attitudes about providing school-age care; and employ- 
ment characteristics, such as salary, responsibilities, and 
job retention. Pre- and post-questionnaires are adminis- 
tered to participating providers and a comparison group 
of providers for this purpose. The third goal is to assess 
the impact of SACC on the overall quality of school- 
age programs, with a particular emphasis on providers’ 
practices and interactions with children (evaluation level 
3), using pre- and post-program observations. Analyses 
of these questions are currently underway, with plans to 
disseminate findings widely. 16 With additional funding, 
the evaluation will expand to include an examination of 
evaluation level 4, i.e., how SACC affects the social and 
academic outcomes of children in school-age programs. 
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Results Level 3: Transfer Level 2: Learning Level I : Reaction 



TABLE 2: THE MULTILEVEL EVALUATION MODEL IN PRACTICE: EXAMPLES FROM THE FIELD 



This table illustrates the four-level method of evaluating professional development through examples from out-of-school time (OST) professional develop- 
ment initiatives. This method can assess outcomes for OST providers participating in professional development, their programs, and the youth they serve. 



Evaluation Level 
and Purpose 



• Understanding OST pro- 
viders’ needs (e.g., skills, 
knowledge, etc.) 

• Assessing participating 
providers’ satisfaction 

• Improving professional 
development initiative 
from providers’ perspec- 
tives and increasing 
participation 



OST Professional 
Development 
Initiatives 



* Bridging the Gap 

* TASC 

* Numerous other 
unpublished initiatives 



Sample Evaluation 
Questions 



* How do staff and ad- 
ministrators perceive 
training workshops? 

• What additional 
training is needed? 



Sample Evaluation 
Methods and 
Sources 



* Surveys of workshop 
participants (pre/post 
or post only) 

* Trainer surveys 

* Participating provid- 
ers’ self-assessments 

* Annual program co- 
ordinator surveys 



Select Evaluation 
Findings and Challenges 



Findings 

• Participating OST providers perceived workshops to 
be useful 

• Trainers reported greater knowledge of participant 
needs 

• Trainers reported workshop improvement due to 
feedback 



* Assessing whether par- 
ticipating OST providers 
learned new information 
from the initiative 
■ Determining how the 
initiative worked, how it 
could be improved, and 
whether it should be 
continued 



■ BEST 

1 Girls Inc. Strong, 
Smart and Bold 

■ MOST 

1 New York State 
School-Age Care 
Credentialing Project 
(in progress) 

1 San Jose 4 Quality 



* Did youth workers 
have greater knowl- 
edge of youth devel- 
opment principles 
after a multi-session 
training program? 

* Did program direc- 
tors have a clearer 
understanding of the 
national program 
mission following a 
leadership institute? 



• Interviews 

• Surveys of participat- 
ing providers 

• Diaries kept by 
providers during the 
professional develop- 
ment initiative and 
their OST programs 



Findings 

• Front-line staff reported increased knowledge of 
youth development after workshops and college 
courses 

• Providers experienced a heightened sense of profes- 
sional affiliation 

• Program directors reported increased confidence, 
enthusiasm, and understanding of program mission 

• Staff recounted “broadened horizons” 

Challenges 

• Front-line staff reported needing more training in 
youth development 



* Assessing whether par- 
ticipating OST providers 
applied the information 
from the initiative in their 
programs 

• Determining how the 
initiative worked, how it 
could be improved, and 
whether it should be 
continued 



■ BEST 

1 Girls Inc. Strong, 
Smart and Bold 

■ MOST 

1 North Carolina 
Quality Enhancement 
Initiative 

1 New York State 
School-Age Care 
Credentialing Project 
(in progress) 

• San Jose 4 Quality 



■ Did front-line provid- 
ers plan and imple- 
ment age-appropriate 
activities after youth 
development training, 
and did they involve 
program youth in 
activity planning? 

■ Did staff and youth 
develop more posi- 
tive relationships? 

* Did the overall 
program environment 
improve? 

* Was there an in- 
crease in networking 
and communication 
among program di- 
rectors after attend- 
ing a multi-session 
leadership institute? 



• Interviews 

• Surveys of participat- 
ing providers 

• Observations 

• Case study method 

• Site portfolios 
(program records and 
self-assessments) 

• Diaries kept by 
providers during the 
professional develop- 
ment initiative and 
their OST programs 



Findings 

• Staff implemented more varied programming and 
increased youth’s active involvement in program 
planning 

• Participating providers were more likely to integrate 
youth development principles into program planning 

• Program directors used more intentional planning 
strategies 

• OST programs’ capacity to serve traditionally under- 
served populations increased 

• Staff-child relationships improved 

• Program environments were rated as more positive 

• Providers communicated more effectively 

• Youth workers reported more support (e.g., 
mentoring and supervision) and networking 

Challenges 

• Program directors benefited more than front-line 
providers 

• Improved knowledge of youth development was not 
integrated into practices by front-line staff 

• Front-line staff needed more ongoing training in 
youth development and appropriate activity structure 

• Staff turnover continued 

• Teamwork skills did not improve 

• Staff members were reluctant to enroll in college 
courses, frequently did not complete courses, faced 
time and financial constraints, and often found that 
course completion did not result in higher salaries 



1 



• Assessing whether the 
initiative improved youth 
outcomes 

• Determining whether 
the program should be 
continued 



1 New York State 
School-Age Care 
Credentialing Project 
(in planning stage) 



1 Did youth’s social 
outcomes improve? 
(planned) 

1 Did youth’s academic 
outcomes improve? 
(planned) 



1 Not available 



1 Not available 
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Implications for Future Evaluation 
and Practice 

Our review confirms that professional development 
initiatives are associated with positive outcomes, includ- 
ing providers’ satisfaction and use of a positive youth 
development framework. Also of note, several of these 
evaluation reports highlighted the role of professional 
development in improving opportunities for networking, 
information sharing, and social support among provid- 
ers. However, to date, evaluations of OST professional 
development efforts have often been limited to the first 
evaluation level — participants’ immediate reports of sat- 
isfaction with the training (Killion, 1999; Mizell, 2003). 
The true prevalence of these satisfaction surveys is not 
known because most are unpublished and used internally 
by programs; however, our scan of the field reveals them 
to be a very common practice. This first evaluation level 
is an important level for learning; training programs and 
trainers can gain valuable information by attending to 
the voices of the providers they serve. However, to really 
understand the benefits of professional development ef- 
forts, programs need to engage in more comprehensive 
evaluations that will address issues of knowledge and skill 
building, best practice, and effectiveness. 

To facilitate this goal and to demonstrate the important 
role that professional development plays in contributing 
to quality OST programs for young people, we make the 
following recommendations for future evaluation. These 
recommendations apply to all professional development 
initiatives, but some are more critical at specific evalua- 
tion levels, as noted below. 

Plan the evaluation before the professional development 
program starts. Evaluation planning should begin at the 
same time as program planning. First, this improves the 
program by encouraging planners to develop a theory 
of how the program will affect outcomes. Second, it 
improves the quality of evaluation by providing the op- 
portunity to collect baseline, or pre-program, measures 
of the intended outcomes. Incorporating this information 
into data analyses makes for a stronger research design. 
This recommendation is particularly important for evalu- 
ation levels 2, 3, and 4. 

Plan a timeline of evaluation. Effectively assessing all 
four levels of outcomes requires strong organizational 
skills, a specific timeline outlining when each aspect of 
the process will occur, and a commitment to following 
this plan (Mizell, 2003). 

Use “backward planning. ” Evaluator Thomas Guskey 
(2001) highlights the importance of starting with the de- 
sired outcomes for youth and working backward to de- 
sign activities that reflect these goals. This is particularly 
important in trying to affect level 4 (youth) outcomes. 
Many professional development programs use the oppo- 



site method, starting with activities requested by provid- 
ers, even if there is no proof that these activities are linked 
with outcomes (Killion, 1999). While the latter method 
may score highly on satisfaction, it may not affect youth 
outcomes. According to staff development expert Hayes 
Mizell (2003), “It is what students need rather than what 
adults want that should shape educators’ learning.” 

Choose outcome indicators based on the goals of the 
program. Outcome data at levels 2, 3, and 4 should be 
collected differently depending on the goals of the pro- 
gram. If the goal is to increase OST providers’ tutoring 
skills, indicators might include providers’ knowledge 
of tutoring strategies, as well as students’ end-of-year 
grades. If the goal of the program is to increase positive 
relationships between students and staff, indicators might 
include staffers’ and youth’s communication skills and 
reports of mutual trust. Evaluators should consider using 
multiple outcome indicators, particularly if the program 
has multiple goals. 

Collect data from appropriate sources. After choosing 
indicators, the next decision should concern the appro- 
priate sources: From whom should the outcome data 
be collected? Data can be collected from OST provid- 
ers, program youth, parents, school records, and other 
sources. The choice of data source largely depends on 
the evaluation level. For example, to measure satisfaction 
and knowledge (levels 1 and 2), evaluators should collect 
data from OST providers themselves. To measure youth 
outcomes (level 4), data should be collected from youth’s 
own reports and also from archival documents, such as 
school records and police reports. (For more information 
on choosing appropriate sources and data collection mea- 
sures, see our Out-of-School Time Evaluation Snapshots , 
available online at www.gse.harvard.edu/hfrp/projects/ 
afterschool/resources.) 

Assess long-term outcomes. The process of developing 
and applying knowledge takes time; the results of pro- 
fessional development programs may not be apparent 
immediately after program completion. Accordingly, 
evaluations should assess outcomes for providers’ prac- 
tices (level 3) and youth (level 4) over multiple years 
(Guskey, 2000). 

Base professional development activities and evalua- 
tions on strong theory and research. As in the field of 
education, there is a growing push for OST programs 
and practices that are based on rigorous research stud- 
ies, which the government refers to as scientifically based 
research (Bouffard, 2003). Theory and research from 
youth development fields provide a reliable framework 
for understanding both what works and why. As such, 
research-driven practices are an important complement 
to the personal experiences and expertise that OST 
professionals bring to bear on staff training. They are 



HARVARD FAMILY RESEARCH PROJECT 



9 



also an important part of establishing how to evaluate 
professional development programs, since evaluations 
should assess only those outcomes they intend to affect. 
Although the use of research-based trainings is increasing 
(e.g., the BEST initiative), it is not yet the norm across 
staff training programs. 

Just as professional development initiatives play a key 
role in ensuring quality out-of-school time program- 
ming, evaluation plays a key role in ensuring effective 
professional development initiatives. Evaluation results 
can be used to understand and build on the strengths and 
weaknesses of these initiatives, to meet accountability 
requirements, and to advocate for more investment in 
professional development. As a result, they can help to 
establish a more coherent and consistent framework for 
professional development activities, including a common 
language and set of goals. This common framework has 
the power to bring information — and quality — to all 
professional development programs, regardless of size, 
location, or focus. 

Suzanne Bouffard, Research Analyst 
Priscilla M. D. Little, Associate Director 
and Project Manager 
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Notes 

1 In this brief we use the terms OST provider and youth ivorker 
to refer to individuals who work with young people, ages 5-18, 
in programs that occur outside of school hours. These out-of- 
school time (OST) programs include before and after school 
programs, tutoring and mentoring programs, and community- 
based youth development programs. 

2 This publication is designed to focus on evaluation rather than 
to provide a comprehensive portrait of professional development 
programs; for readers interested in professional development 
resources, a list is provided on page 11. 

2 For more information on the New York City Beacons Initia- 
tive, visit the website of the Fund for the City of New York at 
www. fcny. org/html/y outh/ school . htm . 

4 For more information on LA’s BEST, visit their website at www. 
lasbest.org. 

5 For more information on the San Francisco Beacons Initiative, 
visit the website of the Community Network for Youth Develop- 
ment at www.cnyd.org. 



6 Our database contains profiles of OST program evaluations, 
which are searchable on a wide range of criteria. It is available in 
the OST section of the HFRP website at www.gse.harvard.edu/ 
hfrp/projects/afterschool/evaldatabase.html. 

7 For more information on TASC, visit their website at www. 
tascorp.org. 

8 For more information on Bridging the Gap, contact Dishon 
Mills at 617-635-1578 or dmills@boston.kl2.ma.us. 

9 MOST was a collaborative effort by many stakeholders from 
1994 to 2001, facilitated by the Wallace Foundation, the Na- 
tional Institute on Out-of- School Time, the Center for Career 
Development in Early Care and Education at Wheelock College, 
and local stakeholders in Boston, Chicago, and Seattle. 

10 San Jose 4 Quality was a collaboration between the National 
Institute on Out-of-School Time, the California School-Age 
Consortium, San Jose area service providers, and the David 
and Lucile Packard Foundation. 

1 1 The Strong, Smart and Bold initiative was conducted from 1993 
to 1996 with support from the Wallace Foundation. 

12 Comparing data before the program — at baseline — and after 
the program can establish whether providers’ knowledge and 
practices did in fact change over the period of the program. 

13 Qualitative data provide descriptive details, often collected from 
a purposive sample of interviews, focus groups, or observations. 
In contrast, quantitative data provide objective, numerical data 
that can be quantified. 

14 BEST was a 1996 national pilot program, funded by the Wallace 
Foundation. 

15 The New York SACC evaluation is supported by funds from 
the Cooperative State Research, Education and Extension Ser- 
vice and the New York State Office of Children and Family 
Services. 

16 For more information on the SACC evaluation, contact Lisa 
McCabe, Ph.D., Evaluation Project Manager, at lam4@ 
cornell.edu. 
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SELECT PROFESSIONAL DEVELOPMENT RESOURCES 



Achieve Boston, a partnership of local public and private 
organizations, provides a Web-based clearinghouse of out- 
of-school time (OST) professional development trainings, 
seminars, college courses, and other offerings in the Boston, 
Massachusetts, area, www.achieveboston.org 

Bringing Yourself to Work: Caregiving in After-School 
Environments is a program run by researchers at Wellesley 
College, which uses a research-driven framework to pro- 
mote reflective practice and emotional intelligence among 
OST providers. For more information on the project and on 
the recent book written by project directors Michelle Selig- 
son and Patricia Jahoda Stahl, see the program’s website at 
www.bringingyourselftowork.com. 

Harvard Family Research Project’s Out-of-School 
Time Program Evaluation Database is a compilation 
of profiles of evaluations of OST programs and initiatives. It 
provides accessible information about evaluation work of both 
large and small OST programs to support the development 
of high quality evaluations and programs in the out-of-school 
time field, www.gse.harvard.edu/hfrp/projects/afterschool/ 
evaldatabase.html 

The Massachusetts School-Age Coalition (MSAC) has 

developed a set of Core Competencies for Massachusetts 
School-Age Practitioners through the PASS (Professional 
Advancement of School-Age Staff) Initiative. MSAC has also 
conducted training workshops across the state, posted a regis- 
try of professional development opportunities, and advocated 
for a career ladder system, www.mass-sac.org 

The National Afterschool Association (formerly the 
National School-Age Care Alliance) Standards for Quality 
School-Age Care include standards for OST staff members 
among their six categories of program quality standards. 
www.naaweb.org 

The National Collaboration for Youth of the National 
Assembly of Health and Human Service Organizations has 
produced a set of Youth Development Worker Competen- 
cies. The competencies are “the knowledge, skills, and per- 
sonal attributes needed by entry-level youth development 
workers to support the critical features of positive youth 
development settings.” www.nydic.org/nydic/documents/ 
FinalCompetenciesl.doc (Word file) 

The National Collaboration for Youth’s special initiative, the 
National Youth Development Learning Network, is “de- 
signed to build and strengthen youth workers’ performance 
by analyzing, synthesizing, and linking existing information and 
training.” Information and professional development resources 
(including a professional development e-newsletter) are avail- 
able at www.nassembly.org/nassembly/2003/initiatives.htm. 

The National Institute on Out-of-School Time (NIOST) 

at Wellesley College is a national voice for promoting a skilled 
and stable workforce. NIOST provides both in-the-field train- 
ings and biannual seminars for OST staff members of all levels, 



publications about the OST workforce and the field in general, 
and other resources, www.niost.org 

The National Staff Development Council (NSDC) 

provides extensive resources on professional development. 
Although focused primarily on education, many of the re- 
sources are also helpful for the OST field. They are available 
at www.nsdc.org. See also the fall 2003 issue of NSDC’s Journal 
of Staff Development, which focuses on evaluating professional 
development. 

A fall 2004 volume of New Directions for Youth Develop- 
ment will focus on professional development in the youth 
development field. Edited by Pam Garza, Lynne Borden, and 
Kirk Astroth, the volume will contain articles on the history 
and current state of professional development initiatives, the 
roles of various organizations and stakeholders, and recom- 
mendations for the field. 

The North Central Regional Educational Laboratory 

(NCREL) has created a National Awards Program for Model 
Professional Development to recognize schools with highly 
effective staff development programs. NCREL has carefully 
examined the programs and practices of these schools in order 
to compile a list of best practices in professional development. 
NCREL has also created a professional development toolkit 
that includes worksheets for planning and conducting all stages 
of the professional development process, including evaluation. 
Although focused on education, this resource can be used for 
OST as well, www.ncrel.org/pd/toolkit.htm 

The Out-of-School Time Resource Center is housed in 
the University of Pennsylvania’s Center for Research on Youth 
and Social Policy. The center aims to help connect OST programs 
with resources, and features professional development tools 
and research. The center’s resources include the recent publi- 
cation, Out-of-School Time (OST) Professional Development Work- 
shops: An Evaluation Framework, www.ssw.upenn.edu/ostrc 

The Partnership for After School Education is a “net- 
work of professionals ... committed to quality education in 
after school settings” and offers professional development 
resources, www.pasesetter.com 

The After-School Corporation (TASC) has produced a 
toolkit. Building the Skills of After-School Staff, which includes 
worksheets for planning, evaluation, and more. It is available 
at www.tascorp.org/publications/catalog/indrep_n. TASC has 
also published a series of resource briefs on staffing issues, 
which are available at www.tascorp.org/publications/catalog. 

YouthNet of Greater Kansas City focuses on promoting 
a set of youth worker standards, providing training and other 
professional development opportunities, and establishing a 
monitoring and assessment process. The organization has 
established a set of standards for providers who work with 
school-age children and another set for those who work 
with teens. Information on these standards can be found at 
www.kcyouthnet.org/developing_standards.asp. Also visit 
YouthNet ’s homepage at www.kcyouthnet.org. 
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